Adapting Populations of Agents

We control a population of interacting software agents. The agents have a strategy, and receive a payoff for executing that strategy. Unsuccessful agents become extinct. We investigate the repercussions of maintaining a diversity of agents. There is often no economic rationale for this. If maintaining diversity is to be successful, i.e. without lowering too much the payoff for the non-endangered strategies, it has to go on forever, because the non-endangered strategies still get a good payoff, so that they continue to thrive, and continue to endanger the endangered strategies. This is not sustainable if the number of endangered ones is of the same order as the number of non-endangered ones. We also discuss niches, islands. Finally, we combine learning as adaptation of individual agents with learning via selection in a population. 1 Populations Of Software Agents In this paper we study a population of software agents [9] that interact with each other. By drawing an analogy between the evolution of software agents and evolution in nature, we are able to use replicator dynamics [14] as a model. Replicator dynamics, first developed to understand the evolution of animal populations, has recently been used in evolutionary game theory to analyze the dynamical behaviour of agents playing a game. Agents playing a game are a good model of software agents when the latter have to make decisions. In replicator dynamics, the mechanism of reproduction is linked to the success or utility of the agents in the interaction with other agents. We think that this process also occurs among software agents. This paper adopts this premise, and then goes on to investigate whether it pays off for a population to retain some unsuccessful strategies as an “insurance policy” against changes in the environment. Each agent is uniquely determined by its code, just as a living organism is determined by its genetic code. For agents, there is no distinction between phenotype and genotype. Consider n different types of agents. At time t, there are pi(t) agents with code i in the population. Just as an agent is determined by i, a population is determined at time t by pi(t), i = 1, . . . , n. The frequency of agent i in the population is xi(t) = pi(t) ∑n i=1 pi(t) . (1) Abbreviate ∑n i=1 pi(t) = p, where p is the total population. Denote the state of the population of agents by x(t) = (x1(t), . . . , xn(t)). Now make the following basic assumptions using terminology widely adapted in evolutionary game theory [14]. Assumption 1 (Game) If agents of type i interact with a population in state x, all agents of type i together receive a payoff u(e,x). Assumption 2 (Replicator Dynamics) The rate of change of the number of agents of type i is proportional to the number of agents of type i and the total payoff they receive: ṗi(t) = pi(t)u(e,x(t)). (2) The proportionality constant in (2) can be absorbed in u. These assumptions are discussed in the rest of this section. In assumption 1, the code i of an agent is identified with a pure strategy e in a game. The notation should not distract the reader, i could have been used instead of e. Identification of a code with a strategy is familiar from evolutionary genetics [12]. During replication of individuals in a population, information is transmitted via DNA. It is straightforward to identify the code of an agent with DNA. Assumption 1, introducing a payoff to software agents, is part of the modelling of software agents as economic agents. Economic principles have been used before in distributed problem solving [8], but in [6] the author has made a start with the analysis of general software agents as economic agents. This paper is part of that project. The replicator dynamics (2) describe asexual reproduction. Agents do sometimes result out of the combination of code from “parent” agents, but such innovative combinations do not occur very often. On a timescale of one year, the replication of agents will be far more important than the reproduction via combination of parent programs. In addition to DNA exchange, our species also passes information between individuals via cultural inheritance. This tends to result in behaviour that is a close copy to the behaviour of the “cultural” parent. If agents are to represent humans in an evolving society, they will also exhibit cultural inheritance or social learning, which follows assumption 2 [7]. In biological systems, one can distinguish long term macroevolution [13], and shorter term microevolution [12]. Assumption 2 can be situated in the field of microevolution. On an even shorter timescale, psychologists observe reinforcement learning. Although the results of reinforcement learning are not passed on to offspring (central dogma of neo-Darwinism), it is possible to cast this learning as replicator dynamics [2]. This adds to the credibility of assumption 2, because software agents will often use reinforcement learning together with replication of code [5]. Biological organisms as well as software agents live in an environment. This is actually the same environment, because software agents act for humans, who live in the biological environment. In the model (2), the change of the environment will be reflected in the change of the payoff function u(e,x), which has to be written u(e,x(t), t) to make the time dependence explicit. It is very important to be able to model this change in environment, because a successful strategy or agent type should be as robust as possible against changes in environment. Another type of evolution that software agents have in common with biological agents is mutation. Strategies should be evolutionary stable if they are to survive mutations [12]. However, mutations can positively contribute to the evolution of a system. Current models tend to concentrate on stochastically perturbing the choice of strategies used [7, 3], rather than the random creation of new strategies. Much work still needs to be done in this area. 2 The Burden Of Maintaining Diversity Agents are pieces of software that act on behalf of humans. Software has a lifetime, so have agents, and humans in a population. The human lifetime is not necessarily the biological lifetime, it may be the time that a human operates in a certain environment. Unsuccessful strategies will die out. This is an essential part of the model (2). Recently there has been much interest in biodiversity [11] and sustainable development [10]. As in all population dynamics, one can ask the question whether it is worth artificially maintaining strategies (or agent types) with a low payoff. The research on biodiversity suggests that this is worthwhile indeed. An agent type is a number i ∈ K = {1, . . . , n}. The set Ke of endangered agent types is defined by Ke = {i ∈ K|u(ei,x) < a}, they have a payoff lower than a. The set Ke will change in time, but the threshold a is fixed. To indicate that the payoffs have been changed, we will use q instead of p for the population, and y instead of x for the frequencies. Assume now that a is the minimum payoff required to sustain a viable population. It is now possible to redistribute the payoff between the non-endangered strategies outside Ke, and the endangered ones in Ke in the following way u(e,x) = a, i ∈ Ke, u(e,x) = u(ei,x)− ∑ j∈Ke [a− u(e ,x)] q − |Ke| , i 6∈ Ke. (3) This transformation conserves the total payoff ∑